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ABSTRACT 

This is a report cf the development of a 
7';cttlty-Course Questionnaire (FCQ) evaluation instrunent at the 
university of Colorado. The evaluation process is conputer based and 
is similar in several ways to that used for processing and reporting 
the results of the Strong Vocational Interest Blank at the University 
of Colorado. A computerized feedback systesi was developed which 
allowed for considerable flexibility. In addition to the availability 
of practically an unlimited number of optional questions that are 
individually preprinted by coaputer, the chief advantage of this 
procedure is that each instructor of a course receives detailed and 
cofiprehensive infornation that is individually specific and at the 
sane tine allows comparisons with various norm groups within the 
university. The instructor also receives subjective responses by 
students on the backside of the questionnaire which asks for the nost 
and least effective aspects of the course and utilizes the critical 
incidence technique. There is a two page feedback system which gives 
one page of numerical and normative feedback while the second page is 
designed to give a verbal report (page 2 is still under development) • 
This is another example of how the use of the computer has opened the 
door to a new dimension in innovation in evaluation. (Author) 
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Robert D. Vfhfttstone 
University of Colorado 

The Idea of evaluating courses and instructors at the college level 
is not new. Although such evaluations have been used for a long tine they «re 
criticised frequently as: 

l« too expensive 

2. lacking i*^ floxihility and adaptability 

3. focusing o^ items that measure popularity or personality 

4. containing negatively worded itens 

5. lacking understandable or appropriate output or feedback 

6. returned to the instructor too late for use the following tern 

7. not related to innovative teaching situations 

8. lacking in statistical analysis of data or overloaded with 
statistical concerns 

9. lacking in appropriate nomative eonparisons 
10* lacking in impact on teaching effectiveness 

This is a report of the development of a Faculty-Course Questionnaire 
(FCQ> evaluation instrument at the University of Colorado. Approximately 
three years ago» the University Conmittee on Effective Teaching asked the 
Student Life Center to develop and administer the f acultycourse evaluation 
program. The University toainer was given the specific responsibility for 
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dttveloplng the Instrument based on certain cooputariaed techniques he had 
already developed for use with the Strong Vocational Interest Blank. The 
Student Life Center director. Dr. Sonald Taylor* also contributed signifi- 
cantly to the developmental process. After writing, sorting, and revising 
a huge pool of items, a preliminary draft was prepared which was carefully 
evaluated by both students and faculty. A stnall pilot project was carried out. 

The new copyrighted questionnaire was first used in the fall of 1972 
with 25,000 questionnalrea scored for about 300 faculty oeobers. Both the 
questionnaire and the computerized feedback were enthusiastically received 
and in the spring of 1973, 43,000 questionnaires were evaluated for over 
650 faculty menbers end 1200 courses. Two questions were added to the core 
question list and several questions were reworded slightly before the fall 
of 1973. 

A cooputerized feedback system was developed which allowed for consider- 
able flexibility. In addition to the availabUity of practically an unlimited 
number of optional questions that are individually pre-prlnted by computer, 
the chief advantage of this procedure is that each instructor of a course re- 
ceives detailed and comprehensive information that is individually specific 
and at the same time allows comparisions with various norm groups within the 
University. The instructor also receives subjective responses by students on 
the backside of the questionnaire which asks for the most and least effective 
aspects of the course and utilizes the critical incidence technique. 

The feedback system designed for University of Colorado faculty consists 
of two pages. Page one features a summary statement of the content of each 
4uestion followed by an A-E average alphabetical rating, a frequency tally of 
the responses to each question (A-4.0 which is positive and E-0.0 which is 
negative, with not-applicable and blank responses also tallied but deleted 
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from further analysis) , departmental average scores and standard deviations 
for each question, a minus or plus mark noting how each faculty member's 
rating differs from the average of ail departmental participants, the aver- 
age score on each question for the faculty member, a minus or plus mark 
noting how each faculty member's rating differs from the average of all depart- 
mental pa^acipants, the average score on each question for the faculty member, 
a minus or plus mark noting how each faculty member's rating differs from 
the total participants of the university and finally the average score and 
standard deviation for each question based on the total university participants. 

lu addition to this basic Information on all 27 core questions, there 
is a tally of responses and an average score for each of up to 16 optional 
questions selected individually by each faculty member from the 162 available 
questions. These optional questions are pr^n-^nted by computer for each 
specific course. 

This single page of output has served the needs of the faculty for four 
semesters and it has received very positive and enthusiastic support. The 
cost of the Faculty Course evaluation was cut in half, the flexibility and 
adaptability was Increased fantastically, the items were focused entirely on 
teaching effectiveness rather than instructof popularity, all items were 
stated positively, the computerized feedback contained both a simple alpha- 
betical rating for those who dislike numerical analysis and adequate numerical 
data for most of the statistically oriented faculty as well as helpful 
student conm»ents for those who claim evaluations cannot be objectified. The 
turn-around time was less than three weeks which included the development 
of all norms, the analysis and processing of data, the packaging and distri- 
bution of results. The teachers of unique courses or those teachers using 
unique procedures had the chance to supply their own optional questions. The 
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instructor In the extra large or extra small course or In the lower division 
or graduate level course could relate his results to courses comparable 
in size or level. Finally, the results can be related directly to teaching 
effectiveness and used both as a diagnostic tool and for the improvement 
of teaching. 

The twenty-seven core questions have been assigned to five subscales 
by subjective procedures which were then revised slightly on a basis of a 
statistical analysis. The five subscales were labeled: I. Effectiveness, 
II, Communication, III. Student Contact, IV. Assignments and V. Evaluation. 
Each 9ub8cale score has high face validity. Each subscale, except I?,, has 
a cronback alpha of .800 to .877. Subscale number IV. was .643. The five 
subscales are processed like the core questions above except no tally is 
given of the response frequencies. In addition, the five subscales plus 
the last two core questions have been norraed for six subgroups at the 
University of Colorado and the average scores; are presented for further 
comparisons. The six subgroups are 1) lower division, 2) upper division, 
3) graduate, 4) small classes, 5) medium classes, and 6) Urge classes. For 
further evaluation and understanding of the results, the number of courses 
and number of students Involved in the departmental and total University 
average scores is presented at the bottom of the page. 

Thus, page one allows a quick summary of the F.C.Q. results by simply 
reading the A-E ratings of the five subscale^ and/or looking a., the number 
of pluses and minuses received when these results are compared with the 
departmental participants and with the total number of university participants. 
For a more detailed analysis of the results, the individual questions can 
be investigated in depth and the subscales as well as questions 26 and 27 can 
be compared to the six subgroups within the tmtversity. The tally of response 
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frequencies and the standard deviations allows for further specific informa- 
tion about results. Some faculty have asked for tests of statistical differ- 
ences and a factoring of the subacalea but most of the faculty have rejected 
these two statistical procedures as unnecessary overkill of the questionnaire data. 



Page two of computerized feedback system is being developed to meet 
the needs of the faculty members who have little knowledge or use for statistical 
reports. It contains a triple profile that gives a pf.cture of the relationship 
of each question's results for the individual faculty member compared to both 
the department and university participants averages. The 27 core questions 
and five subscales are ranked in high-to-low order for easier interpretation. 

The remainder of the page two feedback consists of written comnents 
about the results of the five subscale scores. A written explanation of a 
score was found to be easier to understand and greatly appreciated by faculty 
members who have difficulty interpreting the first page of numerical and 
statistical feedback. Following the interpretative coiaaents, the faculty 
member is encoutaged to investigate the specific questions that make up 
each subscale to determine whether the rating was consistent across all 
questions or unduly Influerxed by one or two specific questions. The pattern 
of responses is also emphasized and possible interpretations are suggested. 

Page ti-fo also allows the possibility of promoting training seminars 
or workshops held by the Committee on Effective Teaching and encourages 
those with outstanding results to volunteer to help present such workshops. 

Thus the P.C.Q. program presented above has overcome the ten most fre^ 
quent criticisms given to the instruments aval) able in this field. 
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Further development Is underway and a few modifications are being considered 
which is another aspect of the flexibility of this paradigm. The availability 
of computer resources and mass data processing systems has opened the door 
to innovation in evaluation, I predict we are standing on the threshold 
of a fantastic breakthrough in this area as our testing and computer skills 
come together to cteate a new approach to an old p em. 



ERLC 



